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colleagues  and  friends  about  the  lack  of  software  expertise  in  weapon 
system  program  offices.  The  purpose  of  this  research  was  to  find  out 
exactly  what  type  of  software  support  existed  in  the  program  offices,  and 
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Abstract 

Research  and  development  of  new  weapon  systems  can  be  very 
expensive.  In  addition,  the  maintenance  of  these  new  systems  can  be 
expensive,  adding  to  the  overall  life  cycle  cost  of  the  systems.  Proper 
design  of  the  new  systems  can  lessen  the  maintenance  costs.  However, 
more  and  more,  weapon  systems  are  not  just  a  product  of  hardware 
engineering.  Computer  systems  and  software  help  to  run  the  systems  more 
efficiently  and  make  it  easier  for  humans  to  control  the  sophisticated 
hardware. 

The  Air  Force  Systems  Command  is  currently  the  main  procurer  of  new 
weapons  systems  for  the  Air  Force.  In  particular,  the  Aeronautical 
Systems  Division  (ASD)  buys  aircraft  and  aircraft  systems  for  the  Air 
Force.  Currently,  ASD  has  a  ’’pool"  of  engineers  for  program  offices  to 
draw  upon  whenever  hardware  technical  expertise  is  needed,  but  the  pool 
of  software  experts  is  very  small.  Do  the  majority  of  the  program 
offices  need  such  a  pool  of  software  experts,  and  if  they  do,  what  type 
of  educational  and  work  experience  should  these  people  have? 

To  answer  this  question,  the  views  of  people  who  are  in  the  program 
offices  were  collected.  Program  managers  and  software  personnel  were 
asked  to  give  their  views  on  education,  training,  and  work  experience. 

The  overwhelming  response  is  that  the  program  offices  do  need 
software  experts,  and  that  even  though  they  have  someone  in  the  office 
who  is  the  designated  software  person,  the  program  offices  still  do  not 
have  enough  support  in  the  management  of  software.  Additionally,  the 
recommended  educational  background  is  in  a  computer  engineering 


discipline.  Computer  engineering  combines  education  in  both  the  hardware 
and  the  software  of  a  computer.  This  education  should  be  enhanced  with 
training  in  systems  acquisition  and  computer  resources  acquisition 
management.  If  possible,  many  of  the  program  offices  would  like  their 
personnel  to  come  to  the  program  with  prior  program  office  experience. 


SOFTWARE  MANAGEMENT  FOR 
WEAPON  SYSTEM  PROGRAMS 


I.  Introduction 


Research  and  development  of  new  weapon  systems  can  be  very 
expensive.  In  addition,  the  maintenance  of  these  new  systems  can  be 


expensive,  adding  to  the  overall  life  cycle  cost  of  the  systems.  Proper 
overall  design  of  Air  Force  weapon  systems  helps  to  hold  down  maintenance 
costs  later  in  the  life  of  those  systems.  In  order  to  ensure  proper 
design,  the  Air  Force  uses  a  staff  of  engineers  to  evaluate  and  monitor 
the  design  of  its  research  and  development  projects.  However,  more  and 
more,  weapon  systems  are  not  just  a  product  of  hardware  engineering. 
Software  is  increasingly  playing  a  part  in  weapon  systems.  According  to 
James  Canan,  senior  editor  of  Air  Force  Magazine,  ten  years  ago,  the  B-1A 
bomber  used  500,000  lines  of  code,  less  than  half  of  the  1,200,000  lines 
of  code  in  the  current  B-1B  (10:46).  Computer  systems  and  software  help 
to  run  the  systems  more  efficiently  and  make  it  easier  for  humans  to 
control  the  sophisticated  hardware.  Good  software  engineering  and 
management  should  also  help  bring  down  the  system  maintenance  costs,  but 
is  the  Air  Force  doing  enough  to  ensure  that  good  design  is  used  in  the 
development  of  weapon  system  software? 


Specific  Problem 

The  Air  Force  Systems  Command  is  currently  the  main  procurer  of  new 
weapon  systems  for  the  Air  Force.  In  particular,  the  Aeronautical 
Systems  Division  (ASD)  buys  aircraft  and  aircraft  systems  for  the  Air 
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Force.  Currently,  ASD  has  a  "pool"  of  engineers  for  program  offices  to 
draw  upon  whenever  hardware  technical  expertise  is  needed.  These 
engineers  include  electrical  engineers,  mechanical  engineers,  chemical 
engineers,  aeronautical  engineers,  and  others  who  help  with  the  struc¬ 
tural  design  of  the  aircraft.  In  addition,  there  are  some  software  and 
computer  systems  experts.  However,  there  are  not  enough  available 
software  personnel  for  all  of  the  program  offices  to  use.  To  make  up  for 
the  lack  of  personnel,  ASD's  Deputy  for  Engineering  educates  its  hardware 
engineers  in  computer  systems  and  software.  Using  a  limited  amount  of 
training,  these  engineers  act  as  technical  representatives  for  computer 
systems  and  software  to  the  program  offices  (5).  With  the  continually 
increasing  number  of  software  systems  used  on  aircraft,  is  there  a  need 
for  software  personnel  in  the  program  offices,  and  if  there  is,  what  type 
of  educational  background  and  work  experience  should  these  people  have? 

Justification 

In  1980,  the  annual  cost  of  software  was  approximately  40  billion 
dollars  (7:17).  In  1976,  Dr  Barry  Boehm  stated  that  compared  to  the  cost 
of  hardware,  software  costs  were  going  to  grow  as  depicted  in  Figure  1 
(7:17-18).  If  Dr  Boehm's  prediction  was  even  partially  true,  software 
costs  have  risen  to  a  level  that  should  cause  concern. 

In  a  joint  memorandum,  dated  15  October  1985,  the  Air  Force  Chief  of 
Staff  and  the  Secretary  of  the  Air  Force  stated  that  the  Air  Force  spent 
about  $3  billion  each  year  on  mission-critical  software  (22).  This 
memorandum  was  written  as  a  result  of  an  Air  Force  study  of  software 


MCCR  include  automatic  data  processing  equipment  and  services  that  are 
generally  involved  with: 

-  intelligence  activities; 

-  cryptologic  activities  related  to  national  security; 

-  command  and  control  of  military  forces; 

-  equipment  that  is  an  integral  part  of  a  weapon  or  weapon  system, 
or; 

-  is  critical  to  direct  fulfillment  of  military  or  intelligence 
missions  (23:9). 

MCCR  does  not  include  automatic  data  processing  equipment  and  services 
used  for  "routine  administrative  and  business  applications  such  as 
payroll,  finance,  logistics,  or  personnel  management"  (20:6). 

Manpower  constraints  will  not  be  covered.  Instead,  the  need  for 
software  personnel  in  program  offices  and  their  qualifications  is  the 
focus  of  this  research.  Whether  or  not  the  Air  Force  has  the  required 
number  of  personnel  is  beyond  the  scope  this  research. 

In  addition,  the  programs  that  will  be  studied  will  be  limited  to 
programs  at  the  Aeronautical  Systems  Division  at  Wright- Patterson  Air 
Force  Base,  Ohio.  Due  to  the  diversity  of  aircraft  systems  handled  at 
ASD,  hardware  concerns  will  not  be  addressed. 

Research  Questions 

There  are  six  research  questions  associated  with  this  thesis: 

1.  What  proportion  of  the  overall  program  costs  include  software 
costs? 

2.  How  many  of  the  program  offices  have  some  kind  of  software 
expertise  resident  in  the  office? 

3.  If  the  office  does  not  have  a  software  expert  in  the  office,  who 


does  the  job  of  ensuring  good  software  design? 


Figure  1.  Hardware/ Software  Cost  Trends 


management  called  "Project  Boldstroke."  According  to  General  Charles 
Gabriel  and  Secretary  Vern  Orr, 


The  study  confirmed  that  our  software  management  expertise  has  not 
kept  pace  with  the  rapid  spread  and  technological  advances  in 
computer-based  weapon  and  information  systems ....  Our  newest  weapons 
and  information  systems — the  B-l,  Peacekeeper,  SACDIN,  and  the  Phase 
IV  Standard  Base  Supply  System — depend  upon  sophisticated  software. 
In  a  very  real  sense,  our  ability  to  deliver  and  support  this 
software  in  a  timely  and  cost-effective  manner  provides  the  Air 
Force  its  most  significant  technological  edge  over  our  potential 
adversaries.  (22) 

This  research  takes  the  next  logical  step  in  that  its  purpose  is  to 
find  out  what  type  of  software  support  is  already  in  place  in  the  program 
offices. 


Scope  of  the  Research 

Because  of  the  two  classifications  of  software,  and  the  different 
regulations  governing  the  acquisition  of  each  classification,  this 
research  will  be  limited  to  Mission  Critical  Computer  Resources  (MCCR). 


4.  What  other  tasks  do  these  people  have? 

5.  What  type  of  educational  background  and  work  experience  do  the 
software  personnel  have? 

6.  What  other  education  and  experience  might  be  needed? 

Summary 

With  the  amount  of  software  needed  on  weapon  systems  now,  it  is 
important  for  the  Air  Force  to  ensure  that  the  contractor-developed 
software  is  properly  managed  early  in  the  program.  Having  well-qualified 
personnel  with  the  appropriate  education  and  experience  may  help  to  keep 
the  costs  of  software  down.  The  purpose  of  this  research  is  to  find  out 
if  those  people  are  needed,  what  support  already  exists  and  who  those 
people  are,  and  what  type  of  education  and  experience  they  should  have. 
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II.  Software  Management 

This  chapter  contains  information  resulting  from  a  literature 
review.  It  is  presented  to  provide  background  information  concerning  the 
need  for  software  management  in  Department  of  Defense  (DoD)  weapon  system 
programs.  First,  the  obstacles  to  effective  software  management  are 
presented.  Then,  there  is  a  discussion  of  how  the  DoD  and  the  Air  Force 
is  trying  to  overcome  these  obstacles.  Finally,  other  topics  that  are 
important  for  software  management  are  briefly  covered. 

Terms  Defined 

Before  going  into  the  discussion  of  software  management,  some  terms 
which  are  basic  to  the  subject  need  to  be  defined. 

Hardware  is  defined  in  Webster's  (26:637)  as  "heavy  military 
equipment  or  its  parts,"  and  as  "the  mechanical,  magnetic,  and  electronic 
design,  structure,  and  devices  of  a  computer."  In  this  paper,  hardware 
will  include  computers  and  the  weapon  systems  (such  as  aircraft)  that 
carry  devices  that  use  software.  These  devices  can  include  equipment 
such  as  aircraft  avionics  and  missile  guidance  systems. 

Software  includes  the  programs,  data,  and  routines  that  are  used  by 
a  computer  to  perform  various  functions  (26:1353).  The  programs,  data, 
and  routines  provide  instructions  to  the  computer  which  tell  the  computer 
what  steps  should  be  taken  when  encountering  certain  prescribed 
conditions.  The  software  may  be  stored  on  magnetic  disk,  tape,  or  in  the 
main  memory  of  the  computer. 

Software  Engineering  is  a  discipline  concerned  with  "the  development 
and  utilization  of  methodologies  and  techniques  for  designing,  imple- 
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meriting,  and  maintaining  software  systems"  (21:11).  In  other  words, 


software  engineering  involves  the  use  of  some  kind  of  pre-defined  plan 
and  method  for  designing  and  developing  a  software  system.  The  method 
may  be  a  new  one  just  developed,  or  one  of  the  many  existing  software 
development  methodologies. 

Software  Management  can  be  broken  down  into  its  two  component  parts. 
Software  was  defined  above,  and  management  can  be  defined  as  "the  act, 
art,  or  manner  of  managing,  or  handling,  controlling,  directing,  etc." 
(26:859).  From  these  two  definitions,  software  management  can  be  defined 
as  the  act  of  controlling  programs,  data,  and  routines  used  by  a 
computer,  so  that  they  are  made  useful  to  man. 

Obstacles  to  Effective  Software  Management 

In  addition  to  the  growth  of  the  need  for  software,  there  are  other 
problems  that  underlie  the  expanding  costs  of  mission-critical  software. 
The  following  problems  presented  highlight  those  problems  that  have 
caused  concern  in  the  DoD. 

Dwindling  Resources.  As  the  need  for  software  grows,  so  does  the 
need  for  software  personnel.  Unfortunately,  the  number  of  trained 
personnel  falls  behind  the  number  needed,  and  as  time  goes  on,  the  gap 
will  widen  (25:92,43).  In  1985,  the  National  Science  Foundation  reported 
that  the  data  processing  community  was  short  between  115,000  and  140,000 
systems  analysts  and  programmers,  and  the  DoD  predicts  that  it  will  be 
short  one  million  software  personnel  by  1990  (24:18).  Since  the  civilian 
sector  also  has  the  same  problem,  this  fact  may  actually  be  worsening  the 
problem  for  the  defense  department,  as  the  civilian  companies  are  hiring 
personnel  away  from  the  government  (10:48). 


Visibility  and  Planning.  Unfortunately,  even  after  approximately  30 
years  in  the  computer  era,  many  senior  managers  are  not  very  well 
educated  about  software  (23:17).  This  lack  of  knowledge  can  lead  to 
problems  with  the  visibility  and  planning  of  the  program.  Software  is 
not  easily  observable,  like  hardware,  and,  as  a  result,  can  be  overlooked 
by  senior  management  (23:17). 

In  addition  to  visibility  problems,  lack  of  sufficient  planning  for 
software  projects  can  increase  costs.  Software  requirements  and 
specifications  are  usually  poorly  defined,  and  therefore  changes  are 
inherent  as  the  project  progresses.  These  changes  are  considered  the 
most  significant  cause  of  cost  increases  in  software  projects  (24:18). 

Lack  of  Standard  Development  Process.  There  is  no  standard 
development  process  for  software.  A  single  requirement  may  result  in  as 
many  solutions  as  there  are  programmers  who  try  to  satisfy  the  require¬ 
ment  (10:50).  Software  development  is  more  of  a  creative  enterprise  than 
it  is  an  engineering  discipline  (24:18). 

Lack  of  Standardization  in  Programming  Language  and  Hardware. 
Contributing  to  the  lack  of  a  standard  development  process  is  the  number 
of  diverse  programming  languages  and  hardware.  The  Department  of  Defense 
alone  uses  more  than  400  of  the  estimated  5000  computer  languages 
(24:18).  The  lack  of  standardization  leads  to  increased  training  costs 
for  software  personnel. 

Strategy  to  Overcome  the  Obstacles 

The  Department  of  Defense  has  instituted  different  programs  and 
written  guidelines  to  try  to  solve  the  problems  mentioned  above.  A  few 
of  the  programs  and  guidelines  are  presented  here. 
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Software  Engineering  Institute.  In  December  1984,  the  Department  of 
Defense  awarded  a  S103  million,  five-year  contract  to  Carnegie-Mellon 
University  to  create  a  Software  Engineering  Institute  (SEI)  (24:18).  The 
SEI  was  set  up  to  "investigate  the  entire  software  engineering  scene  in 
the  U.S.  to  foster  new  software  technologies,  and  ...  to  expedite  the 
transition  of  software  technologies  into  practice"  (10:49).  The  SEI  is 
accomplishing  this  goal  by  recruiting  corporations  and  universities  to  be 
affiliates  in  the  hopes  that  collaboration  of  the  "top  experts"  in  the 
practice  will  evelop  useful  software  engineering  tools  (24:19). 

Software  Technology  for  Adaptable  and  Reliable  Systems.  The 
Software  Technology  for  Adaptable  and  Reliable  Systems  (STARS)  initiative 
was  a  result  of  a  commitment  made  by  the  Department  of  Defense  to 
Congress  to  institute  a  software  technology  initiative  (25:43-44).  The 
STARS  initiative  was  instituted  in  1983,  and  is  supposed  to  last  for 
seven  years,  going  through  four  separate  stages  in  that  time  (25:44). 

The  overall  STARS  goal  is  to  improve  productivity  while 
achieving  greater  system  reliability  and  adaptability  (in  the  face 
of  increasingly  demanding  requirements)  through  software  development 
and  in-service  support  processes  that  are  more  responsive, 
predictable,  and  cost-effective.  STARS  is  concerned  with  improving 
both  the  product  (e.g.,  latent  defects  per  thousand  lines  of  code) 
and  the  process  (e.g.,  productivity).  (12:25) 

Project  Bold  Stroke.  In  late  1985,  as  a  result  of  an  Air  Staff 
study  on  software  management,  the  Air  Force  instituted  Project  Bold 
Stroke  (21).  Bold  Stroke  is  a  "plan  to  improve  software  management 
expertise"  (16:1-2).  The  project  has  four  main  objectives: 


I.  Create  overall  Air  Force  awareness  of  the  criticality  of 
software  and  computer  based  technology  to  information  and  mission 
critical  systems. 


II.  Provide  software  training  and  education  necessary  to  ensure 
that  the  Air  Force  fully  uses  the  potential  available  through 
software  and  computer-based  technology. 

III.  Survey  Air  Force  software  perse  -nel  requirements  and  develop  a 
more  comprehensive  approach  to  recruiting  and  managing  military  and 
civilians  in  this  career  field. 

IV.  Project  future  software  personnel  requirements  with  emphasis  on 
the  appropriate  mix  of  officers,  civilians,  NCOs  and  contractor 
support.  (16:1-2) 

Education  and  Training.  Both  the  Air  Force  Institute  of  Technology 
(AFIT)  at  Kright-Patterson  AFB,  Ohio,  and  the  Air  Force  Systems  Command 
Systems  Acquisition  School  (SAS)  at  Brooks  AFB,  Texas,  offer  courses 
pertaining  to  the  acquisition  of  MCCR.  The  courses  are  available  to 
personnel  in  system  program  offices  who  occupy  positions  involved  with 
MCCR  acquisition. 

AFIT  offers  at  least  three  courses  dealing  with  MCCR  acquisition 
(2).  One  course,  entitled  Mission  Critical  Computer  Resources 
Acquisition,  provides  lessons  that  instruct  current  and  prospective 
software  acquisition  managers  on  topics  relative  to  MCCR  acquisition 
(2:71).  The  topics  covered  include  hardware,  software,  the  acquisition 
life  cycle,  quality  assurance,  standards,  test  and  evaluation,  design  and 
development,  and  contract  surveillance  (1). 

SAS  offers  the  Computer  Resources  Acquisition  Course.  This  course 
is  also  targeted  to  personnel  who  are  working  in  a  program  office 
acquiring  MCCR.  The  topics  included  in  this  course  are  the  acquisition 
life  cycle,  contracts,  documentation,  standards,  Ada,  testing,  software 
metrics,  production  and  deployment,  and  maintenance.  (3) 

Both  schools  also  offer  courses  addressing  the  acquisition  life 


cycle  in  general. 


Ada.  In  1975,  a  Department  of  Defense  working  group  defined 
higher-order  programming  language  requirements  and  initiated  the 
development  of  Ada,  after  a  review  of  existing  languages  found  them 
inadequate  (6:42).  Ada  is  the  result  of  an  objective  to  achieve  a 
standard  programming  language  in  the  Department  of  Defense  (23:24).  Ada 
is  different  from  most  other  programming  languages  in  that  it  was 
developed  specifically  for  embedded  computer  systems  (8:4).  In  order  to 
ensure  that  Ada  would  be  a  standard  language  in  the  DoD,  an  interim  list 
of  DoD-approved  higher  order  languages,  which  included  Ada,  was  approved 
as  policy  in  DoD  Directive  5000.31  (8:15).  Additionally,  to  prevent  the 
development  of  many  Ada-like  languages,  "Ada”  was  approved  as  a 
registered  trademark  of  the  DoD  (8:20). 

Program  Office  Staffing  (16:7-3,7-4,7-5,7-7,7-8).  To  help  the 
program  offices  with  their  software  management,  the  Air  Force  has  three 
primary  sources  of  computer  resources  expertise.  The  first  two  areas 
presented  are  career  areas  for  military  officers,  and  the  last  area 
presented  is  a  career  field  for  civilian  personnel  who  work  for  the  Air 
Force. 

First,  there  is  the  communications-computer  systems  career  area, 
which  encompasses  both  the  communications-electronics  and  the  automated 
data  processing  areas.  Those  people  in  this  career  field  provide  program 
formulation,  engineering,  and  acquisition  support  for  the  Air  Force. 
Officers  in  this  career  field  have  an  Air  Force  specialty  code  in  the 
49XX  series.  The  first  two  digits  of  the  code  represent  the  communica¬ 
tions-computer  career  field,  and  the  last  two  digits  represent  the 
specific  specialty  that  the  officer  works  in  under  that  career  field. 
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Secondly,  there  is  the  scientific  and  development  engineering  career 
area.  This  career  area  includes  the  26XX,  the  27XX,  and  the  28XX  series. 
Three  new  specialty  codes  were  added  to  help  manage  computer  resource 
acquisition.  The  codes  are  2625 — Computer  Research  Scientist, 
2736--Computer  Systems  Acquisition  Manager,  and  2885 — Computer  Systems 
Engineer. 

The  final  source  of  expertise  in  computer  resources  is  supplied  by 
civilian  personnel  who  work  for  the  DoD.  These  people  are  identified  by 
the  series  code  GS-334,  or  the  Computer  Specialist  Series.  The  positions 
identified  by  this  series  will  be  responsible  for  "performing  the  design, 
implementation,  maintenance,  and  modification  of  systems  that  use  digital 
computers  to  accomplish  required  work  processes"  (16:7-3).  The  series 
does  not  include  those  positions  which  require  full  qualification  in 
scientific  disciplines  encompassing  mathematics,  engineering,  or  computer 
science. 

A  typical  weapon  system  program  office  should  have  some  people  from 
the  above  career  areas  if  the  weapon  system  involves  computer  resources. 

Other  Guidelines.  To  provide  guidelines  in  acquiring  software,  the 
Department  of  the  Defense,  the  Air  Force,  and  the  Air  Force  Systems 
Command  have  written  regulations  and  standards.  Some  of  these 
regulations  and  standards  include  DoD-STD-2167,  MIL-STD-483A,  AFR  800-14, 
AFSCP  800-43,  and  AFSCP  800-14. 

The  first  standard,  DoD-STD-2167,  is  titled  "Defense  System  Software 


Development,"  and  contains  the  requirements  for  development  of  MCCR.  It 
describes  a  typical  system  life  cycle  and  the  requirements  for  each  phase 
of  the  life  cycle  (13:1).  In  addition,  the  standard  was  developed  to 
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ensure  standardized  documentation  of  all  software  products.  This  stan¬ 
dardization  would  be  beneficial  during  updating  and  maintenance  of  the 
software.  (13) 

Military  standard  483A  establishes  requirements  for  configuration 
management  of  systems,  equipment,  munitions,  and  computer  software 
(17:1).  The  standard  provides  procedures  to  manage  configuration  items 
by  identifying  and  documenting  characteristics  of  the  item,  controlling 
changes  to  those  characteristics,  and  recording  the  status  of  the  item 
(17:5). 

The  Air  Force  description  of  what  reviews  and  audits  should  be  done 
throughout  the  acquisition  life  cycle  of  a  weapon  system  is  contained  in 
AFR  800-14.  The  regulation  is  a  guideline  to  ensure  that  the  software 
portion  of  the  system  is  planned,  developed,  and  acquired  alongside  the 
hardware  (18:1). 

Recognizing  that  the  program  managers  need  help  in  managing  software 
as  well  as  hardware,  a  pamphlet  on  software  management  indicators  was 
developed  by  Air  Force  Systems  Command  in  support  of  Project  Bold  Stroke 
( 14 : i ) .  The  pamphlet,  AFSCP  800-43,  provides  heuristics  and  standards 
that  can  be  used  to  compare  against  the  software  development  management 
figures  provided  by  the  contractor  (14:3). 

Subsequent  to  the  development  of  AFSCP  800-43,  Air  Force  Systems 
Command  also  developed  AFSCP  800-14.  Entitled  "Air  Force  Systems  Command 
Software  Quality  Indicators,"  this  pamphlet  is  meant  to  be  used  in 
conjunction  with  AFSCP  800-43  to  promote  quality  software  development 
(15:3).  The  pamphlet  provides  a  list  of  quality  indicators,  their  normal 
behavior  patterns,  and  guidelines  for  collecting  the  indicators'  values 


This  is  just  a  small  representation  of  the  standards  and  regulations 


provided  to  help  the  Air  Force  manage  its  software  programs  effectively. 

A  list  of  other  standards  and  regulations  is  provided  in  Appendix  A. 

Software  Quality  Assurance. 

Software  quality  assurance  is  a  large  topic  that  has  had  research 
focused  on  it  alone;  however,  it  is  presented  here  for  familiarization 
purposes,  as  it  must  be  taken  into  account  in  software  projects. 

Software  quality  assurance  has  been  described  as  "the  activities 
necessary  to  assess  and  measure  the  quality  of  a  software  product" 
(4:104).  It  has  also  has  been  described  as  a  program  used  to  "detect, 
analyze,  report,  and  correct  software  errors"  (19:109).  In  either  case, 
quality  assurance  in  software  projects  is  important  in  order  to  receive 
software  that  is  usable  when  delivered  by  the  contractor. 

In  Air  Force  programs,  there  are  numerous  activities  during  the  life 
cycle  of  a  program.  These  activities  include  reviews,  test  and 
evaluation,  and  independent  verification  and  validation  (IV&V)  (18).  All 
these  activities  are  planned  and  scheduled  to  help  ensure  that  the 
government  gets  the  product  that  it  wants.  These  activities  also 
contribute  to  the  quality  assurance  program  for  the  software.  A  more 
detailed  explanation  of  the  acquisition  life  cycle  is  contained  in 
Appendix  C. 

Software  Safety  Management. 

The  final  topic  to  be  covered  is  software  safety  management. 

Software  safety  management  can  be  defined  as  "the  ability  of  a  computer 
program  and  its  associated  data  to  avoid  system  faults  that  endanger  life 


or  property"  (11:148).  This  topic  is  very  important  as  the  amount  of 
software  used  to  control  aircraft  systems  grows.  If  an  important  system 


fails,  both  the  life  of  a  pilot  and  the  aircraft  he  is  flying  may  be 
lost.  Software  safety  management  should  be  considered  early  in  the 
project  since  70  percent  of  the  problems  in  software  systems  can  be 
attributed  to  management  and  planning  deficiencies  (11:148). 

Summary 

The  program  manager  must  be  aware  of  the  many  problems  that  may 
endanger  the  success  of  the  program.  Software  is  just  one  of  the  many 
systems  that  are  a  part  of  the  whole  program.  Software  itself  has  its 
own  set  of  problems  that  the  program  manager  has  to  control.  Some  of  the 
obstacles  to  effective  software  management  have  been  presented,  along 
with  some  of  the  strategies  and  guidelines  used  to  overcome  those 
obstacles . 


This  chapter  contains  the  methodology  that  will  be  used  to  answer 
the  research  questions  posed  in  Chapter  one.  The  general  method  used  to 
answer  the  research  questions  involved  data  collection  by  survey.  In 
this  case,  the  vehicle  used  to  survey  will  be  interviews.  The  following 
steps  will  be  done  to  accomplish  the  research. 

1.  Interview  guides  will  be  prepared. 

2.  Program  offices  will  be  selected  based  on  pre-determined 
criteria. 

3.  The  program  managers  of  selected  programs  will  be  interviewed  to 
collect  their  views  on  the  software  costs  of  their  respective 
programs,  and  who  their  software  experts  are  if  they  have  any. 

4.  The  identified  software  experts  themselves  will  be  interviewed, 
and  their  views  on  education  and  job  experience  will  be 
collected. 


Guided  interviews  will  be  used  when  collecting  the  data.  In  a 
guided  interview,  questions  are  prepared  in  advance.  The  same  questions, 
asked  in  the  same  order,  are  used  when  interviewing  each  individual. 

Using  a  guided  interview  ensures  that  each  person  will  be  asked  the  same 
questions  in  the  same  order.  This  helps  prevent  biasing  answers  differ¬ 
ently  because  of  the  previous  question  asked. 

The  questions  for  the  interview  guides  will  be  developed  with  the 
objective  of  answering  the  research  questions.  In  addition,  the  follow¬ 
ing  questions  will  be  asked  when  developing  the  interview  questions: 


1.  Does  the  question  provide  information  within  the  scope  of  this 
research? 

2.  Is  the  question  easily  understood? 

3.  Does  the  question  suggest  a  certain  answer,  or  is  it  open-ended 
enough  to  let  the  individuals  express  their  own  thoughts? 


way. 

3.  The  program  offices  must  be  accessible  to  the  researcher. 

4.  The  program  offices  must  be  willing  to  participate  in  the 
research. 


Using  the  selection  criteria,  the  following  program  offices  are 


chosen: 

1. 

ASD/B1M 

Directorate  of  Projects  for  the 

B-1B 

2. 

ASD/RWN 

Strike  System  Program  Office  (SPO) 

3. 

ASD/YWS 

Strategic  and  Airlift  Simulator 

SPO 

4. 

ASD/AEA 

Director  of  Common  Avionics 

5. 

ASD/AF/C-17A 

Directorate  of  C-17  Projects 

The  selected  program  offices  will  be  contacted  by  letter  asking  for 
their  support.  The  cover  letter  will  briefly  explain  the  research  with 
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copies  of  the  interview  guides  attached.  Sending  the  interview  guides 
will  allow  the  prospective  interviewees  to  prepare,  in  advance,  answers 
to  the  questions.  The  program  offices  will  respond  to  the  letter  with  a 
list  of  volunteers  for  the  interviews.  The  volunteered  personnel  will 
then  be  contacted  to  set  up  appointments  which  are  convenient  for  both 
interviewer  and  interviewee.  An  example  of  the  cover  letter  and  the 
interview  guides  are  contained  in  Appendix  B. 

Interview  of  Program  Managers 

An  interview  will  be  used  since  the  type  of  information  needed  is 
not  readily  available  except  from  the  program  office  itself.  In  order  to 
keep  the  number  of  contact  points  per  program  to  a  minimum,  the  program 
manager  or  designated  substitute  will  be  interviewed. 

The  interview  guide  will  be  used  when  interviewing  the  program 
managers.  The  interviews  will  cover  certain  topics,  which  include: 

a.  software  costs, 

b.  staffing  and  organization, 

c.  duties  and  responsibilities,  and 

d.  education  and  training. 

The  interview  of  the  program  managers  will  answer  the  following 
research  questions.  The  research  questions  are  numbered  with  the  same 
numbers  used  in  the  first  chapter. 

1.  What  proportion  of  the  overall  program  costs  include  software 
costs? 

2.  How  many  of  the  program  offices  have  some  kind  of  software 
expertise  resident  in  the  office? 


3.  If  the  office  does  not  have  a  software  expert  in  the  office,  who 
does  the  job  of  ensuring  good  software  design? 

4.  What  other  tasks  do  these  people  have? 

Questions  to  both  the  program  manager  and  the  software  personnel 
will  address  the  fourth  research  question.  This  will  be  done  to  try  to 
get  as  complete  information  as  possible  concerning  the  duties  of  the 
software  personnel. 

One  of  the  problems  anticipated  is  that  because  of  the  nature  of  the 
program  managers’  jobs,  being  able  to  interview  randomly  selected  program 
managers  would  be  difficult  since  they  are  very  busy  and  travel  often. 

In  order  to  accommodate  this  problem,  a  sample  of  convenience  will  be 
used.  In  anticipation  of  this  problem,  one  of  the  criteria  for  selection 
will  be  that  the  program  office  be  willing  to  participate  in  the 
research.  Interviewing  those  program  managers  who  will  be  accessible 
constitutes  a  sample  of  convenience.  Because  of  the  convenience  sample, 
results  of  the  data  collected  cannot  be  statistically  generalized  to  all 
of  the  programs,  however,  since  all  of  the  programs  have  to  operate  under 
the  same  regulations,  a  logical  generalization  may  be  possible. 

Interview  of  Software  Personnel 

Essential  to  the  performance  of  the  software  personnel  is  the 
training  and  education  that  they  receive.  The  purpose  of  the  interview 
of  the  software  personnel  will  be  to  ascertain  what  type  of  educational 
background  and  work  experience  they  have,  and  what  type  of  education  they 
feel  they  need  to  have  to  perform  their  jobs.  In  addition,  questions 
concerning  what  type  of  work  they  actually  do  in  the  program  office  will 
be  asked.  The  interviews  will  be  done  using  the  interview  guide,  and 


only  those  personnel  working  for  interviewed  program  managers  will  be 
interviewed.  If  the  program  office  does  not  have  what  they  consider 
software  personnel,  those  personnel  doing  the  job  of  software  management 
will  be  interviewed,  since  their  views  on  education  and  training  will 
help  answer  the  research  questions. 

The  interview  questions  will  cover  the  following  topics: 

a.  formal  education, 

b.  education  and  training  received  during  employment, 

c.  former  job  experience,  and 

d.  recommendations  for  training. 

The  research  questions  to  be  answered  from  the  interviews  of 
software  personnel  include  the  following  questions.  Again,  to  be  con¬ 
sistent,  the  questions  are  numbered  using  the  same  numbering  system  as 
the  first  chapter. 

4.  What  other  tasks  do  these  people  have? 

5.  What  type  of  educational  background  and  work  experience  do  the 
software  personnel  have? 

6.  What  other  education  and  experience  might  be  needed? 

Data  Analysis 

Since  the  nature  of  the  overall  problem  statement  is  subjective,  an 
actual  statistical  test  is  not  appropriate.  Rather,  the  answers  to  the 
research  questions  will  be  categorized  and  summaries  of  the  different 
answers  will  be  presented.  None  of  the  answers  will  be  attributed  to  any 
individual.  Rather,  the  answers  will  be  combined  to  present  a  picture  of 


the  interviewees  as  a  whole. 


Software  Costs.  Because  of  the  large  differences  in  size  and 


structure  of  the  chosen  program  offices,  direct  comparison  of  software 
costs  is  not  appropriate.  If  software  costs  are  a  large  percentage  of  a 
small  program,  the  software  part  of  the  program  deserves  visibility  since 
it  can  cause  significant  program  cost  overruns.  Overall  software  costs 
per  program  will  be  analyzed  in  addition  to  percentages  since,  in  the 
case  of  large  programs,  a  small  percentage  of  overall  program  costs  may 
still  be  a  significant  amount  of  money. 

Software  Expertise.  As  discussed  earlier,  the  Air  Force  has  three 
main  areas  or  sources  of  computer  resources  expertise — the 
communications-computer  systems  career  area  and  the  development 
engineering  career  for  officers,  and  the  computer  specialist  series  for 
the  civilian  personnel.  Because  of  the  different  sources  of  software 
expertise  available,  no  one  type  of  source  will  be  singled  out  as  the 
resource  to  have  in  the  program  office.  Instead,  the  program  managers 
will  be  asked  if  they  have  any  software  personnel,  and  it  will  be  left  to 
the  program  manager  to  decide  what  they  consider  a  software  expert.  If 
they  respond  that  they  indeed  have  someone  fulfilling  the  role  of 
software  personnel,  whether  or  not  they  are  from  one  of  the  three  sources 
described  is  not  an  issue. 

Ensuring  Good  Software  Design.  Since  the  contractors  who  develop 
the  weapon  system  design  the  software  that  resides  on  the  system,  it  is 
the  responsibility  of  the  program  office  to  ensure  that  the  contractor 
does  have  good  software  design.  If  the  program  office  does  have  software 
personnel,  these  people  are  the  ones  who  will  see  that  the  contractor 
provides  well  designed  and  software.  If  not,  the  program  office  will  be 
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asked  who  oversees  the  management  of  the  software  portion  of  the  system 
for  the  program  office. 

Tasks  and  Responsibilities.  To  get  an  idea  of  what  the  overall 
responsibilities  of  the  software  personnel  are,  questions  addressing 
duties  and  responsibilities  will  be  asked  of  both  the  program  managers 
and  the  software  personnel.  The  responses  to  the  questions  will  be 
summarized  and  listed  by  program  office  since  each  office  is  organized 
differently. 

Education.  Training,  and  Work  Experience.  Questions  addressing 
formal  education,  training,  and  work  experience  will  be  asked  of  both  the 
program  managers  and  the  software  personnel.  Results  of  these  questions 
will  be  summarized  and  used  to  answer  the  last  two  research  questions. 


Research  Observations 


This  chapter  contains  the  resulting  views  collected  during 
interviews  with  the  selected  program  offices.  The  collection  and 
analysis  of  data  followed  a  pre-determined  methodology.  After  the 
interview  guides  were  complete,  program  offices  were  selected  and 
contacted.  Those  individuals  who  volunteered  to  be  interviewed  were 
scheduled  for  an  interview  and  the  results  of  those  interviews  were 
analyzed  as  possible  answers  to  the  research  questions.  Each  program 
office,  except  ASD/RWN,  provided  one  person  from  program  management  and 
two  software  personnel.  ASD/RWN  was  only  able  to  provide  one  software 
person.  Therefore,  there  were  a  total  of  fourteen  persons 
interviewed — five  from  program  management,  and  nine  software  personnel. 

The  type  of  weapon  system  a  System  Program  Office  (SPO)  is  procuring 
may  determine  the  organizational  structure  and  the  assignment  of 
personnel  within  that  organization.  Before  discussing  the  results  of  the 
interviews,  a  short  description  of  each  program  office  is  presented.  The 
terms  "program  office"  and  "SPO"  are  used  interchangeably  throughout  the 
rest  of  this  document.  Additionally,  since  all  of  the  program  offices 
are  in  ASD,  they  mav  be  referred  to  by  their  "three-letter"  symbol  by 
dropping  the  "ASD/"  part  of  their  office  symbol. 

ASD/B1,  or  the  B-1B  SPO,  is  what  is  termed  as  a  "super  SPO."  In 
other  words,  the  main  responsibility  of  the  B-1B  SPO  is  the  procurement 
of  one  complete  weapon  system,  in  this  case  the  B-1B  bomber,  and  all  of 


its  support  equipment  and  facilities.  All  members  of  that  organization 


are  dedicated  to  providing  support  in  acquiring  systems  and  subsystems  of 
that  one  aircraft. 

The  Strike  SPO,  ASD/RWN,  is  a  smaller  program  office,  residing  in 
the  same  organization  as  other  small  program  offices.  Its  main 
responsibility  is  to  procure  a  system  that  can  reside  on  one  or  more 
different  types  of  aircraft,  in  support  of  that  aircraft's  overall 
mission.  Personnel  of  the  organization  may  work  on  one  program,  or  share 
time  on  other  small  programs  for  which  the  organization  is  responsible. 

The  third  program  office.  ASD/YWS,  is  also  a  small  program  office, 
residing  in  the  same  organization  as  other  small  program  offices.  The 
main  responsibility  of  this  SPO  is  to  procure  aircrew  training  systems 
and  simulators  that  help  teach  aircrews  to  fly  aircraft  that  are  normally 
assigned  to  Strategic  Air  Command.  Because  the  program  office  is  one  of 
several  in  the  same  organization,  it  too  has  to  share  some  of  its 
personnel  with  other  program  offices  in  that  organization. 

The  Common  Avionics  SPO,  ASD/AEA,  is  the  third  small  program  office 
residing  in  the  same  organization  as  other  program  offices.  This  program 
office  procures  standardized  or  common  systems  that  are  used  on  one  or 
more  different  types  of  aircraft.  The  systems  help  the  aircraft  perform 
their  missions.  AEA  also  shares  some  of  its  personnel  with  other  program 
offices. 

Finally,  the  last  program  office  interviewed,  the  C-17  SPO  can  also 
be  considered  a  "super  SPO."  Although  the  organization  to  which  the  C-17 
SPO  belongs  has  program  offices  for  more  than  one  system,  the  personnel 
assigned  to  this  program  are  separated  from  the  others  by  being  located 
in  a  separate  building.  Currently,  the  program  office  is  in  the  process 


of  becoming  its  own  organization.  It  is  responsible  for  the  procurement 


of  the  017  transport  aircraft  and  all  of  its  associated  support 
equipment . 

Software  Costs 

The  first  research  question  asked,  "What  proportion  of  the  overall 
program  costs  include  software  costs?"  To  get  an  answer  to  this  research 
question,  three  questions  were  asked  of  personnel  working  in  program 
management  at  each  of  the  SPOs.  The  three  questions  were: 

1.  What  do  you  estimate  are  your  software  costs  for  your  program? 

2.  What  is  the  total  cost  of  your  program? 

3.  What  do  you  do  to  ensure  that  software  costs  are  kept  to  a 
minimum? 

Table  1  shows  the  results  of  these  questions  for  each  of  the  program 
offices. 

Table  1.  Results  of  Research  Question  1 


SPO 

Total  Cost 
of  Program  (S) 

Software  Costs 
(Estimated  %  of  Total) 

Estimated 

Software  Costs  (S) 

B-IB 

20.5  Billion 

5-10 

1.03  -  2.05  Billion 

RUN 

3.8  Billion 

1 

45  Million 

YWS 

300  Million 

50  -  70 

200  -  220  Million 

AEA 

100,000/unit 

20  -  30 

20,000  -  30,000 

C-17 

34  Billion 

* 

*  -  Estimates  of  software  costs  could  not  be  given  since  the  program  was 
not  far  enough  into  the  acquisition  life  cycle. 


The  software  costs  presented  are  only  "best  guess"  estimates  given 
by  the  program  managers.  The  contract  types  awarded  to  the  prime 
contractors  were  fixed  price  contracts.  In  other  words,  an  amount  was 
negotiated  between  the  government  and  the  contractor  and  the  contractor 
is  obligated  to  deliver  the  specified  system  for  that  amount  of  money. 
Since  the  price  included  hardware,  software  and  all  other  necessary  items 
such  as  labor  and  administrative  costs,  it  was  not  possible  for  the 
program  managers  to  break  out  exact  costs  for  the  software.  Along  the 
same  lines,  the  managers  did  not  really  have  an  answer  to  the  third 
question  except  to  say  that  since  the  contracts  were  fixed  price,  the 
contractor  was  bound  by  the  contract  to  deliver  the  software  included  in 
the  system  for  the  negotiated  price  of  the  overall  system. 

The  software  costs  do  amount  to  a  large  percentage  of  the  overall 
program  costs  for  two  of  the  program  offices.  For  the  B-lB  SPO  and  RWN, 
however,  even  though  the  percentages  of  the  software  costs  are  lower,  the 
actual  amounts  are  large.  The  estimated  software  cost  for  RWN  is  lower 
than  it  probably  should  be  since  the  $45  million  includes  only  the  cost 
of  manpower  used  to  develop  the  software.  It  does  not  include  the  costs 
for  quality  assurance,  configuration  management,  or  the  costs  for 
changing  the  software. 

Software  Expertise 

To  answer  the  question,  "How  many  of  the  program  offices  have  some 
kind  of  software  expertise  resident  in  the  office,"  the  following  series 
of  questions  were  asked  of  the  program  managers. 
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1.  Do  you  have  software  personnel  in  your  organization?  If  not,  do 
you  need  software  personnel?  Who  do  you  have  doing  the  job  of  the 
software  personnel  in  the  mean  time? 

2.  Where  do  you  use  your  software  personnel  in  your  organization? 

3.  How  many  people  are  permanently  assigned  as  software  personnel? 

4.  How  many  people,  in  total,  are  assigned  to  the  organization? 

The  results  of  these  questions  are  presented  in  Table  2,  All  of  the 

program  offices  interviewed  had  what  they  considered  to  be  software 
personnel.  YWS,  which  is  considered  to  have  a  software  intensive  program 
due  to  the  simulators  it  is  procuring,  had  a  greater  proportion  of 
software  personnel  to  the  overall  total  of  people  assigned  to  the  program 
at  40%.  The  total  number  however,  only  includes  people  directly  assigned 
to  that  program.  Recall  that  the  Strategic  and  Airlift  SPO  is  just  one 
program  office  in  an  organization  of  many.  There  are  other  personnel 
matrixed  in  to  handle  administrative  tasks,  such  as  contracting, 
configuration  control,  and  other  functions. 

Table  2.  Software  Personnel  in  Program  Offices 

Software  Number  of  Software  Total  Number 

SPO  Personnel  (Y/N)  Personnel  of  Personnel  Percentage 


In  all  of  the  SPOs,  the  engineers  and  some  of  the  software  personnel 


i 

* 

*! 
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come  from  the  ASD's  Deputy  for  Engineering  (ASD/EN)  (5).  Therefore,  to 
get  the  required  software  support  personnel,  the  program  office,  must  go 
to  ASD/EN  and  request  support.  The  personnel  may  be  then  assigned  to  the 
SPO’s  organization,  but  are  still  resources  of  ASD/EN.  To  further 
complicate  matters,  in  the  smaller  SPOs  where  there  are  more  than  one 
program  office  in  the  same  organization,  the  engineers  and  software 
personnel  may  be  assigned  work  from  more  than  one  program.  In  all  of  the 
small  SPOs  interviewed,  at  least  one  software  person  was  dedicated  to 
that  program,  and  other  personnel,  who  had  work  from  other  programs,  were 
available  when  needed. 

Ensuring  Good  Software  Design 

The  third  research  question  was,  "If  the  office  does  not  have  a 
software  expert  in  the  office,  who  does  the  job  of  ensuring  good  software 
design?"  Since  all  of  the  program  offices  interviewed  had  someone  that 
they  considered  a  software  expert,  this  research  question  did  not  apply. 
Additionally,  four  of  the  program  offices  interviewed  had  some  kind  of 
outside  support  to  provide  independent  verification  and  validation  (IV&V) 
of  the  software  and/or  its  documentation.  The  C-17  SPO  had  not  yet 
progressed  far  enough  in  the  acquisition  life  cycle  to  have  software 
products  that  needed  IV&V,  and  the  prime  contractor  was  performing 
verification  and  validation  of  the  software  its  subcontractors  had 
completed. 

Tasks  and  Responsibilities 

The  objective  of  the  fourth  research  question  was  to  find  out  what 
other  tasks  the  software  personnel  have.  To  answer  this  question,  the 

28 


.1 


duties  and  responsibilities  of  the  software  personnel  are  presented. 


Since  each  organization  has  its  own  procedures,  the  duties  and 
responsibilities  for  each  SPO  will  be  discussed  separately.  Both  the 
program  managers  and  the  software  personnel  were  asked  questions 
concerning  this  subject.  The  program  management  personnel  were  asked  the 
following  questions: 

1.  What  are  the  duties  of  your  software  personnel? 

2.  Who  reviews  your  software  technical  documents? 

3.  What  type  of  procedures  do  you  have  to  ensure  adequate  review? 

4.  Who  conducts  the  reviews  and  audits  of  the  software? 

The  software  personnel  were  asked  the  following  question  concerning 
their  duties  and  responsibilities: 

1.  What  are  your  duties  in  this  program? 

B-1B  SPO.  The  software  personnel  in  the  B-1B  SPO  are  the  technical 
representatives  of  the  program.  They  initiate,  staff,  and  brief  software 
change  requests  when  needed,  and  oversee  the  contracts  pertaining  to  the 
software  packages  assigned  to  them.  The  review  of  the  technical  docu¬ 
ments  are  done  by  the  software  personnel,  but  because  of  time 
constraints,  only  a  "top-level"  review  is  accomplished. 

The  audits  and  reviews  of  the  software  are  taken  care  of  at  the 
Functional  Configuration  Audit,  the  Physical  Configuration  Audit,  the 
engineering  review  (or  the  Preliminary  Design  Review),  and  the  config¬ 
uration  review  (or  Critical  Design  Review).  The  configuration  and  data 
management  directorate  of  the  SPO  is  responsible  for  settir.j  up  the 
meetings  and  publishing  the  minutes. 


ASD/RWN.  In  the  Strike  SPO,  the  software  personnel  provide  services 
to  include  review  of  the  documentation,  audits,  and  participation  in  the 
testing  of  the  systems.  In  addition  to  the  software  personnel  assigned 
to  the  SPO,  RWN  has  a  contract  with  a  private  company  to  provide 
independent  verification  and  validation.  The  company  provides  in-depth 
review  of  the  technical  documents,  while  the  software  personnel  also 
review  the  documents.  This  procedure  is  also  followed  during  audits. 

ASD/YWS.  YWS  also  has  an  independent  verification  and  validation 
contractor.  Their  job  is  to  review  technical  documents  for  consistency, 
completeness,  and  correctness.  The  software  personnel  in  this  SPO  are 
responsible  for  making  sure  that  the  software  meets  specifications,  and 
that  it  is  designed,  coded,  and  documented  in  accordance  with  standards. 
In  other  words,  the  SPO  personnel  check  for  technical  correctness,  while 
the  IV&V  contractor  checks  the  written  documents  to  make  sure  that  they 
are  written  correctly.  The  software  personnel  are  also  responsible  for 
reviewing  the  software.  They  ensure  that  the  software  is  documented 
according  to  DoD-STD-2167,  and  that  the  software  unit  development  folders 
are  kept  up  to  date. 

The  SPO  uses  a  computer  system  to  track  documents  for  when  they  are 
due  from  the  contractor,  and  where  the  documents  are  in  the  organization 
for  review.  The  configuration  and  data  management  directorate  is 
responsible  for  keeping  the  system  up  to  date.  They  are  also  responsible 
for  conducting  the  functional  and  physical  configuration  audits. 

ASD/AEA.  During  the  interview  with  the  program  manager,  he 
enumerated  the  following  duties  and  responsibilities  of  his  software 
personnel : 
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-  help  write  software  performance  specifications, 

-  identify  what  requirements  need  to  be  allocated  for  software, 

-  help  estimate  software  costs, 

-  help  write  statements  of  work  for  contracts, 

-  provide  technical  advice  to  program  management, 

-  monitor  the  contractor’s  performance  of  the  contract  with  respect  to 
all  software  development  and  testing  activities,  and 

-  participate  in  approving  test  plans  and  test  procedures,  and  in 
evaluating  test  results. 

The  software  personnel  are  responsible  for  reviewing  the  technical 
documents  and  for  running  the  software  audits  and  reviews.  There  is  no 
formal  procedure  for  ensuring  adequate  review,  and  the  personnel  rely  on 
their  training  when  performing  the  reviews. 

C-17  SPO.  The  software  management  support  for  the  C-17  SPO  is 
divided  into  two  different  groups.  There  is  a  group  of  people  who  work 
with  the  program  management.  They  are  not  matrixed  into  the  SPO  from 
ASD/EN,  and  are  the  program  manager’s  experts  on  software.  They  head  up 
the  program  management  team  in  the  software  area  and  are  active  in  the 
computer  resources  working  group.  The  SPO  is  also  procuring  a  management 
information  system  (MIS)  and  these  software  personnel  are  responsible  for 
overseeing  that  process.  The  second  group  of  software  personnel  are 
matrixed  into  the  SPO  from  ASD/EN  and  are  mainly  responsible  for  portions 
of  the  software  that  reside  on  automated  test  equipment. 

Both  groups  are  responsible  for  reviewing  technical  documents,  and 
there  are  no  formal  procedures  for  adequate  review.  An  office  of  primary 
responsibility  is  designated  for  the  review  of  a  document,  and  that 
office  is  held  responsible  for  adequate  review.  Software  reviews  and 


audits  are  held  by  the  prime  contractor  since  the  software  is  subcon¬ 
tracted  out  by  the  prime  contractor.  In  other  words,  the  subcontractor 
writes  the  software  and  delivers  it  to  the  prime  contractor.  The  prime 
contractor  reviews  the  software,  and  the  SPO  personnel  are  allowed  to 
attend  the  reviews.  The  SPO  monitors  the  procedures  and  performance  of 
the  prime  contractor. 

Education,  Training,  and  Work  Experience 

The  fifth  research  question  asked,  "What  type  of  educational 
background  and  work  experience  do  these  software  experts  have?"  To 
answer  this  question,  the  software  personnel  were  asked  six  different 
questions  regarding  their  jobs,  past  and  present,  and  the  education  and 
training  they  had  received.  The  questions  asked  were  as  follows. 

1.  What  other  programs  have  you  worked  on? 

2.  How  much  do  you  rely  on  your  previous  work  experience  in  your 

current  job? 

3.  What  type  of  educational  background  do  you  have? 

4.  How  has  this  education  been  beneficial  to  your  job? 

5.  What  type  of  formal  training  have  your  had  in  connection  with 

your  job? 

6.  How  has  this  training  been  beneficial  to  your  job? 

The  answers  to  the  fourth  and  sixth  questions  are  used  to  form 
recommendations  for  training  and  education.  The  answers  to  the  other 
questions  form  the  basis  for  a  possible  answer  to  the  fifth  research 
question.  Table  3  contains  information  regarding  the  education  and  work 
experience  of  each  person.  It  also  has  their  recommendations  on 
education  and  training. 
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Table  3.  Individual  Views  on  Education,  Training,  and  Work  Experience 


(1) 

Previous 

Work 

Experience 

(2) 

Formal 
Education 
(Note  1. ) 

(3) 

Recommended 
Education 
(Note  2. ) 

(4) 

Recommended 
Training 
(Note  3. ) 

1. 

Private  Industry 

Elec  Engr 

A,  B 

1,2,6 

2. 

None 

Comp  Sci 

B,  C 

1,2,9 

3. 

Avionics  "hot 
bench” 

Engr  Phys 

A,  C 

4, 10 

4. 

Program  Office 

Elec  Engr 

D 

12 

5. 

Program  Office 

Elec  Engr 
(working  on 

M.S.  Elec  Engr) 

F 

1 

6 . 

Program  Office 

Elec  Engr 

D 

1 

7. 

Branch  Chief 
ASD/ENA SC 

B.S.  Elec  Engr 
M.S.  Elec  Engr 
M.S.  Engr  Mgt 

B,  C 

1,3 

8. 

Private  Industry 
Co-op  w/Government 

Engr  Phys 

E 

I, 2, 3, 7, 

II, 13,14 

9. 

Computer  Center 

Comp  Sci 
(working  on 

M.  B.  A. 

A 

1,2, 5, 8 

NOTE  1.  Comp  Sci  -  Computer  Science;  Elec  Engr  -  Electrical  Engineering; 
Engr  Mgt  -  Engineering  Management;  Engr  Phys  -  Engineering  Physics. 

NOTE  2.  A  -  Computer  Engineering;  B  -  Computer  Science  w/hardware 
courses;  C  -  Electrical  Engineering  w/software  courses;  D  -  Engineering; 

E  -  Business  w/minor  in  computers;  F  -  Chemical  Engineering. 

NOTE  3.  1  -  Systems  Acquisition;  2  -  Computer  Resources  Acquisition 

Course;  3  -  Contracting;  4  -  Ada;  5  -  Configuration  Management; 

6  -  DoD-STD-2167;  7  -  Jovial;  8  -  Logistics  Support;  9  -  RADAR; 

10  -  Real-time  Programming;  11  -  Reliability  of  Hardware  and  Software; 

12  -  Software  Engineering;  13  -  Software  Quality  Assurance;  14  -  1750 
Processors. 


Education.  Of  the  nine  software  personnel  interviewed,  two  were 
educated  in  engineering  physics,  five  held  degrees  in  electrical 
engineering,  and  two  people  had  a  degrees  in  computer  science.  In 
addition,  one  of  people  who  held  an  undergraduate  degree  in  electrical 
engineering  held  a  graduate  degree  in  electrical  engineering  and  another 
graduate  degree  in  engineering  management.  Two  others  were  working 
towards  Masters  degrees  in  Business  Administration,  and  one  person  was 
close  to  completing  his  graduate  degree  in  electrical  engineering.  Of 
the  two  people  working  on  their  MBAs,  one  held  a  degree  in  engineering 
physics,  and  the  other  held  a  degree  in  computer  science.  The  person 
working  on  his  graduate  degree  in  electrical  engineering  already  held  a 
baccalaureate  degree  in  the  same  subject. 

Training.  Since  the  interviewees  were  allowed  to  list  the  different 
training  classes  they  had  attended  when  they  answered  the  question 
regarding  training,  there  are  more  than  nine  different  answers.  A 
summary  of  the  different  training  subjects  and  the  number  of  responses 
pertaining  to  each  subject  is  contained  in  Table  4.  The  subjects  are 
ordered  by  the  number  of  responses  pertaining  to  them  from  highest  to 
lowest.  Those  subjects  with  ties  are  ordered  within  themselves  in 
alphabetical  order. 

In  addition  to  the  questions  about  training  asked  of  the  software 
personnel,  the  program  managers  were  asked  what  type  of  training  program 
they  had  for  their  personnel.  All  of  them  replied  that  they  had  no 
formal:'  ted  training  program,  and  that  they  relied  on  ASD/EN  to  supply  all 
the  training  necessary  before  the  engineers  and  software  people  were 
assigned  to  the  SPO.  That  did  not  mean,  however,  that  the  program  office 
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would  not  send  their  people  to  any  training.  If  a  course  was  available, 
and  the  program  office  could  afford  both  the  time  and  the  money,  their 
engineers  and  software  people  were  allowed  to  attend  the  course. 


Table  A.  Training  Attended 


Subj  ect  of  Responses 


Computer  Resources  Acquisition  Course  (CRAC)  5 
Systems  Acquisition  Courses  (SYS  100,  200,  SAS)  5 
Ada  (Programming  Language)  3 
Software  Quality  Assurance  3 
Software  Engineering  2 
Cost  Estimating  1 
DoD-STD-2167  1 
Hardware/Architecture  1 
Jovial  (Programming  Language)  1 
Real-time  Systems  1 
Reliability  1 
System  Software  1 


The  training  that  the  software  personnel,  who  came  from  ASD/EN, 
received  was  not  necessarily  the  same  for  each  person.  Training  periods 
in  EN  lasted  anywhere  from  two  weeks  to  two  years.  On  the  extremes,  one 
person  said  that  he  was  in  EN  for  a  week  or  two  when  a  requirement  for  a 
software  person  came  up,  and  he  was  assigned  to  a  program  office.  The 
other  extreme  involved  a  person  who  spent  two  years  working  on  an 
avionics  "hot  bench."  Those  who  fell  in  the  middle  spent  an  average  of 
six  months  in  EN,  and  attended  the  Computer  Resources  Acquisition  Course, 
among  other  training  courses. 

Not  all  of  the  software  personnel  were  matrixed  into  the  SPOs  from 
EN.  Two  of  the  software  personnel  in  the  C-17  SPO  were  not  from  EN. 
Therefore,  they  did  not  receive  any  of  the  training  that  EN  provides  to 
its  engineers. 
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Work  Experience.  Only  three  of  the  nine  software  personnel 
interviewed  had  previous  program  experience.  Of  those  three,  only  one 
person  did  not  rely  on  the  previous  experience  in  his  current  job.  His 
previous  experience  was  in  source  selection  for  another  program,  and  he 
felt  that  he  never  used  that  experience  when  conducting  his  duties. 

The  two  remaining  persons  were  from  small  SPOs  and  had  been  in  other 
SPOs  within  the  same  organization.  Both  said  they  relied  on  their 
previous  work  experience  in  their  current  job,  but  not  to  the  same 
degree.  The  first  person  said  he  relied  "95”o  on  previous  experience"  in 
doing  his  current  job.  The  other  person  said  he  rarely  needed  his 
previous  engineering  experience,  but  when  he  did,  he  needed  quite  a  bit 
of  his  engineering  expertise.  In  his  own  words,  he  relied  on  his 
previous  experience  "heavily,  but  not  often." 

Of  those  who  had  not  worked  on  other  programs,  four  had  previous 
work  experiences  in  other  than  weapon  system  program  offices.  The  first 
person  had  worked  for  a  short  time  in  the  private  sector  for  a  defense 
contractor.  He  felt  that  this  experience  helped  when  he  prepared 
contract  proposals  and  other  contracting  documents.  The  second  person 
had  some  experience  with  industry  and  "co-op”  experience  with  the 
government  while  attending  college.  He  felt  that  he  had  built  his 
knowledge  from  the  experience  gained  from  those  previous  jobs  since  his 
education  was  in  engineering  physics. 

The  third  person  had  previously  been  a  communications-computer 
officer  and  had  worked  in  a  computer  center  as  a  requirements  and  plans 
officer.  He  felt  that  this  experience  helped  him  in  his  current  job 
since  the  SPO  was  acquiring  a  management  information  system  (MIS).  The 


basic  understanding  of  rules  and  regulations  regarding  the  acquisition  of 


an  '■IIS  as  well  as  buying  computer  equipment  was  some  of  the  knowledge  he 
had  gained  at  the  computer  center. 

Finally,  the  fourth  person  worked  in  an  avionics  facility  on  an  Ada 
avionics  hot  bench  within  ASD/EN  for  two  yours  before  coming  into  his 
present  job.  In  his  previous  job,  he  had  learned  how  to  write  and  test 
software  for  the  same  type  of  hardware  that  is  used  on  avionics 
equipment.  He  felt  that  this  tvpe  of  exoerience  was  "incrediblv 
valuable"  to  him  in  his  current  job.  He  had  only  been  on  his  current  job 
for  six  months  at  the  time  he  was  interviewed,  and  he  felt  that  he  was 
verv  well  "on  top"  of  his  job,  and  that  he  would  not  be  without  the 
experience  he  gained  in  his  previous  job.  He  felt  that  his  education  in 
engineering  physics  had  taught  him  to  approach  each  problem  from  an 
engineering  perspective,  and  that  perspective  helped  him  to  learn  about 
software  easier. 

Education,  Training,  and  work  Experience  Veeded 

The  objective  of  the  last  research  question  was  to  find  out  what 
other  education  and  experience  might  be  needed.  To  answer  the  research 
question,  the  software  personnel  were  asked  the  following  questions. 

1.  what  type  of  training  do  you  feel  you  need  but  have  not  been 

able  to  get? 

2.  What  type  of  education  do  you  recommend  for  your  job? 

3.  What  type  of  previous  work  experience  do  you  recommend? 

4.  What  type  of  training  programs  do  you  recommend? 

The  first  and  fourth  questions  were  asked  to  get  an  idea  of  what 

type  of  training  the  software  personnel  felt  they  needed  to  do  their 
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jobs.  The  other  questions  were  asked  for  the  same  reason  pertaining  to 


their  respective  subjects. 

Education.  In  addition  to  the  question  asked  of  the  software 
personnel,  the  program  managers  were  asked  what  they  recommended  as  a 
good  educational  background  for  their  software  people.  A  summary  of 
their  answers  is  contained  in  Table  5.  The  recommended  education  is 
listed  by  subject  and  the  number  of  responses  pertaining  to  that 
response . 


Table  3.  Recommended  Education 


Subi ect  of  Responses 


Computer  Engineering  6 
Computer  Science  with  Hardware  courses  6 
Electrical  Engineering  w/Software  courses  4 
Engineering  3 
Business  with  minor  in  computers  1 
Chemical  Engineering  1 
Computer  Science  with  Math  1 


NOTE.  The  numbers  do  not  add  up  to  equal  the  fourteen  persons  interviewed 
since  they  were  allowed  to  list  any  number  of  educational  subjects. 


The  computer  engineering  degree  consists  of  courses  in  both  computer 
science  and  electrical  engineering.  Having  an  education  in  both  software 
and  hardware  was  important  to  the  majority  of  those  interviewed.  Nine  of 
the  fourteen  people  interviewed  responded  one  or  more  of  the  top  three 
responses. 

Three  people  responded  that  engineering  was  a  recommended  educa¬ 


tional  background,  while  one  person  responded  that  chemical  engineering 


was  recommended.  All  four  ot  these  people  contended  that  the  discipline 
and  way  of  approaching  a  problem  learned  in  an  engineering  education  was 
enough  background  for  anyone  to  be  a  software  manager.  They  maintained 
that  the  knowledge  in  software  engineering  and  the  other  education  needed 
to  accomplish  the  job  could  be  learned  in  training  available  through  the 
Air  Force  and  other  agencies. 

Training.  In  their  responses  to  recommended  training,  the  inter¬ 
viewees  were  allowed  to  list  any  number  of  training  subjects.  The  sub¬ 
jects  and  the  number  of  responses  for  each  subject  are  listed  in  Table  6. 
Because  of  the  numerous  responses  each  person  gave,  the  total  number  of 
responses  does  not  add  up  to  nine.  The  subjects  are  listed  in  order  of 
the  number  of  responses  given  for  that  subject  from  highest  to  lowest. 


Table  6.  Recommended  Training 


Subject 


Systems  Acquisition 

Computer  Resources  Acquisition  Course  (CRAC) 
Contracting  (Writing  Statements  of  Work,  etc.) 
Ada 

Configuration  Management 
DoD-STD-2167 

Jovial  (Programming  Language) 

Logistics  Support 
RADAR 

Real-time  Programming 

Reliability  of  Hardware  and  Software 

Software  Engineering 

Software  Quality  Assurance 

1750  Processors 


sf  of  Responses 


When  asked  what  type  of  training  they  felt  they  needed,  but  had  not 
been  able  to  get,  there  were  four  responses  for  training  in  the  Ada 
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programming  language,  and  four  responses  for  training  in  systems 


acquisition. 

Work  Experience.  Even  those  people  without  any  previous  work 
experience  were  allowed  to  recommend  what  they  perceived  as  a  good  job  to 
have  before  coming  into  their  current  position.  A  summary  list  of  types 
of  work  experience  follows.  They  are  not  listed  in  any  particular  order. 


-  Working  in  the  private  sector  for  a  defense  contractor 

-  Working  in  another  5P0  on  another  program 

-  Getting  "hands-on"  experience  with  developing  computer  systems  on  an 
avionics  hot  bench 


-  Working  in  an  organization  considered  to  be  a  prospective  user  of 
the  system  that  the  SPO  is  procuring 

-  Being  senior  in  rank  with  previous  program  office  experience 

-  Working  on  other  small  programs  from  start  to  finish 


Having  previous  experience  in  engineering,  a  computer  center,  a 
program  control  office,  and  configuration  management 


Other  Observations 


One  of  the  questions  that  was  asked  of  the  software  personnel,  but 
did  not  really  fall  under  just  one  of  the  research  questions  was,  "How 
much  knowledge  in  software  engineering  is  needed  to  accomplish  your 
responsibilities?"  Three  of  the  software  personnel  responded  that  they 
needed  their  knowledge  of  software  engineering  to  understand  what  the 
contractor  was  doing,  and  to  understand  documentation.  The  rest 
responded  that  they  needed  a  little  knowledge  of  software  engineering, 
but  were  really  learning  what  to  do  in  their  job  through  experience. 

Another  observation  made  was  that  the  two  software  personnel  who  did 
not  have  a  degree  in  either  electrical  engineering  or  computer  science, 
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recommended  and  desired  to  take  courses  in  computer  hardware  and 
programming  languages.  (See  columns  2  and  U  of  Table  3.) 

Third,  even  though  the  SPOs  had  software  personnel  assigned,  all  of 
the  program  managers  felt  that  they  could  use  more  software  personnel. 

The  C-17  SPO  was  in  the  process  of  expanding  the  number  of  software 
personnel  and  the  number  of  hardware  engineers.  The  program  management 
knew  that  they  would  need  more  than  three  people  to  manage  the 
acquisition  of  all  of  the  software  related  to  a  large  transport  aircraft. 

The  last  observation  made  was  that,  of  the  three  sources  of  computer 
resources  personnel,  only  two  of  the  nine  personnel  came  from  any  of 
those  sources.  Both  of  them  came  from  one  source  in  particular,  which  is 
included  in  the  development  engineering  career  area.  They  both  had  the 
specialty  code  of  2625 — Computer  Research  Scientist.  The  other  people 
also  came  from  the  development  engineering  career  area,  but  did  not  have 
the  specialty  codes  identified  as  a  source  of  computer  resources 
expertise.  None  of  the  software  personnel  interviewed  were  from  the 
communications-computer  systems  career  area  or  the  civilian  computer 
specialist  series  at  the  time  of  the  interviews.  However,  one  of  the 
Computer  Research  Scientists  was  previously  a  communications-computer 
systems  officer. 

Finally,  even  though  this  was  not  considered  to  be  an  observation, 
those  people  who  were  interviewed  provided  additional  comments.  They 
expressed  concern  over  the  length  of  training  since  the  SPOs  cannot 
afford  to  have  personnel  gone  from  the  office  to  attend  three  weeks  of 
training.  Additional  comments  are  provided  in  Appendix  D. 


V. 


Conclusions  and  Recommendations 


In  order  to  tie  the  research  to  some  useful  form  of  information, 
this  final  chapter  presents  conclusions  to  the  research  questions  and 
recommendations  for  improvement.  This  research  does  not  provide  an 
overall  solution  to  the  questions.  Rather  it  presents  one  contribution 
to  the  solution. 

Research  Summary 

Even  though  the  SPOs  were  of  different  sizes,  and  were  responsible 
for  programs  ranging  from  avionics  to  total  aircraft  systems,  they  were 
bounded  by  the  same  rules  and  regulations.  Because  of  these  rules  and 
regulations,  the  SPOs  operated  similarly.  They  did  differ  in  those  cases 
where  different  programs  determined  costs,  and  different  management 
styles  determined  different  delineation  of  responsibilities.  The 
following  is  a  short  summary  of  the  research  and  a  conclusion  for  each  of 
the  research  questions. 

Software  Costs.  The  software  costs  did  amount  to  large  percentages 
in  two  of  the  programs.  For  anotner  two,  even  though  the  percentages 
were  not  as  large,  the  estimated  costs  were  large  sums  of  money  in 
millions  and  billions  of  dollars.  Software  can  no  longer  be  ignored. 

The  cost  of  software  may  eventually  determine  the  price  tags  of  weapon 
systems . 

Because  of  the  nature  of  the  type  of  contracts  that  the  SPOs  were 
using,  they  were  not  able  to  control  the  cost  of  the  software  separately. 
As  long  as  the  contractor  delivered  the  required  system  for  the 
negotiated  price,  the  contract  would  be  satisfied. 


Software  Expertise.  All  of  the  program  offices  had  someone  that  was 


considered  a  software  expert.  The  majority  of  these  people  were  engi¬ 
neers  with  some  training  in  software. 

ASD/YKS,  the  Strategic  and  Airlift  Simulator  SPO,  had  the  largest 
proportion  of  software  costs,  and  software  personnel.  This  large 
proportion  was  because  of  the  nature  of  the  program.  The  SPO  was 
responsible  for  acquiring  a  simulator  that  would  be  used  to  train 
aircrews  of  a  specific  aircraft.  Simulator  systems  are  software 
intensive  since  it  is  the  software  that  simulates  the  different 
conditions  that  aircraft  encounter. 

The  Common  Avionics  SPO,  ASD/AEA,  had  the  next  largest  proportion  of 
software  costs,  but  did  not  have  a  large  percentage  of  software 
personnel.  Its  percentage  was  close  to,  or  the  same  as,  the  remaining 
SPOs. 

Even  though  the  program  offices  realized  the  importance  of  software 
in  their  respective  systems,  they  were  not  staffing  a  large  proportion  of 
software  personnel  in  the  offices.  This  understaffing  may  be  attributed 
to  the  shortage  of  qualified  software  personnel  in  the  Air  Force. 

Ensuring  Good  Software  Design.  All  of  the  program  offices  had 
software  personnel  who  monitored  the  contractor’s  performance  of  their 
contracts.  The  contractors  were  responsible  for  good  software  design. 

The  program  offices  were  concerned  with  ensuring  that  the  software 
performed  according  to  specifications.  It  seems  that  no  one  in  the 
piogram  offices  had  the  time  or  personnel  available  to  ensure  good 
software  design. 
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Tasks  and  Responsibilities.  Even  though  some  of  the  duties  and 


responsibilities  differed  from  SPO  to  SPO,  all  of  their  software 
personnel  were  responsible  for  reviewing  the  technical  documents 
pertaining  to  the  software  portions  of  their  systems.  They  made  sure 
that  the  documents  and  .software  items  were  delivered  on  time  and 
according  to  technical  specifications,  and  they  attended  the  reviews  of 
the  software. 

Education,  Training,  and  Work  Experience.  The  majority  of  the 
software  personnel  interviewed  had  educational  degrees  in  electrical 
engineering  with  some  courses  in  software.  Additionally,  most  of  them 
had  some  training  in  systems  acquisition.  Finally,  only  three  of  them 
had  previous  SPO  experience. 

Of  those  people  who  came  to  the  program  offices  from  ASD/EN,  their 
training  program  in  EN  was  not  consistent  among  them.  The  amount  of  time 
spent  in  EN  varied  along  with  the  type  of  training  each  person  received. 

Education,  Training,  and  Work  Experience  Needed.  Most  of  the 
program  managers  and  software  personnel  interviewed  recommended  that 
education  in  both  the  hardware  and  software  of  computers  is  the  best 
background  for  the  job  of  software  management.  The  most  recommended 
training  courses  included  systems  acquisition  courses  and  the  Computer 
Resources  Acquisition  Course.  Finally,  most  of  the  software  personnel 
said  that  previous  program  office  experience  would  have  helped  them  in 
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their  current  jobs. 


Recommendations 


These  recommendations  were  compiled  after  analyzing  what  type  of 
education,  training,  and  work  experience  was  most  recommended  by  the 
program  managers  and  software  personnel  interviewed. 

Education.  The  recommended  educational  background  for  the  software 
personnel  is  an  education  in  both  the  hardware  and  software  of  a  computer 
system.  This  type  of  education  is  usually  called  computer  engineering 
and  consists  of  courses  in  software  engineering  and  computer  architec¬ 
ture.  Understanding  both  the  softxvare  and  the  computer  system  on  which 
it  runs  will  help  the  software  person  to  better  evaluate  the  specifi¬ 
cations,  documentation,  and  test  reports  of  the  computer  resources. 

Training.  The  recommended  training  includes  courses  in  both  the 
acquisition  of  computer  resources  and  system  acquisition.  The  courses  in 
computer  resource  acquisition  would  familiarize  the  software  personnel  in 
documentation,  contract  management,  and  details  that  pertain  only  to 
computer  resources.  The  system  acquisition  courses  would  familiarize  the 
software  personnel  with  the  acquisition  life  cycle,  and  enable  them  to 
see  how  the  software  portion  of  the  program  fits  in  with  the  acquisition 
of  the  whole  weapon  system. 

Additional  training  should  be  available  to  the  software  personnel 
after  being  assigned  to  a  program  office.  These  courses  will  help  them 
keep  up  with  the  changing  technology.  These  courses  should  not  be  longer 
than  five  days  in  duration,  since  most  program  offices  cannot  afford  to 
have-  personnel  gone  too  long  for  training. 

Work  Experience.  If  possible,  before  being  assigned  to  a  program 
office,  the  software  personnel  should  have  some  experience  in  working  in 


a  program  office.  This  could  be  done  while  the  personnel  are  still  being 
trained  at  ASD/EN.  They  could  be  temporarily  assigned  with  an  exper¬ 
ienced  person  to  work  on  a  program  for  a  short  period  of  time.  This 
would  provide  some  insight  as  to  how  a  program  office  operates. 

All  of  these  recommendations  are  possible  only  if  the  time, 
training,  and  personnel  are  available,  and  many  times  the  opposite  is 
true.  However,  if  the  Air  Force  is  interested  in  having  good  software 
management  in  the  program  offices,  these  recommendations,  based  on 
current  views,  are  what  the  Air  Force  should  follow. 

Problems  Encountered 

As  anticipated,  coordinating  times  when  program  office  personnel 
would  be  available  for  interview  was  very  difficult.  Most  of  the 
software  personnel  were  available,  but  a  few  were  out  of  town  when  their 
offices  were  contacted.  The  program  managers  were  a  bit  more  difficult 
contact.  rour  of  the  five  were  out  of  town  quite  a  bit,  or  had  other 
meetings  to  schedule  around  when  they  were  in  town.  This  difficulty  in 
setting  up  interview  appointments  extended  the  total  amount  of  time 
needed  to  interview  all  of  the  people.  Even  though  there  were  only 
fourteen  people  interviewed  and  each  interview  lasted  from  twenty  minutes 
to  an  hour,  the  total  time  to  complete  all  of  the  interviews  took  over 
six  weeks. 

Secondly,  because  software  is  not  priced  separately  in  the 
contracts,  it  was  very  difficult  for  the  program  offices  to  offer 
anything  but  best  guess  estimates  of  the  software  costs.  This  problem 
led  to  difficulty  in  accurately  assessing  the  proportion  of  program  costs 
dedicated  to  software. 


Further  Research 


Further  research  into  the  management  of  software  and  software  costs 
is  recommended.  Approaches  to  research  in  these  two  areas  are  discussed. 

Management  of  Software.  Further  research  into  what  the  software 
personnel  actually  have  to  do  on  a  day  to  day  basis  is  recommended.  A 
listing  of  all  of  the  softxvare  personnel  could  be  obtained  from  ASD/EN 
and  a  random  sample  could  be  generated.  A  survey  instrument  in  the  form 
of  a  questionnaire  could  be  sent  to  the  sample.  The  questionnaire  would 
contain  detailed  questions  asking  about  time  spent  with  contracting 
documents,  software  documentation,  and  actually  looking  at  programming 
code.  Other  questions  regarding  knowledge  needed  in  hardware  and 
software  could  be  asked.  And  finally,  questions  pertaining  to  decision 
making  on  matters  concerning  computer  resources  could  be  asked.  From  the 
answers  to  the  questions,  more  accurate  recommendations  regarding 
educational  background  and  training  could  be  made. 

Software  Costs.  Until  the  cost  of  software  is  broken  out 
separately,  it  will  be  difficult  to  do  any  more  research  on  the  costs. 
However,  once  the  costs  are  broken  out,  analysis  and  comparisons  between 
different  program  offices  and  how  they  control  the  costs  of  software 
could  be  a  basis  for  research. 

Conclusion 

Once  the  Air  Force  understands  how  much  of  its  weapon  systems  depend 
on  software,  and  develops  the  resources  to  address  the  growing  depen¬ 
dence,  it  can  begin  to  control  the  cost  of  software.  Research  into  those 
resources  can  help  identify  how  the  Air  Force  can  approach  the  problem. 


This  research  addressed  personnel  resources — whether  or  not  there  was  a 
need  for  them,  and  what  type  of  education  and  work  experience  they  should 
have.  Personnel  with  a  computer  engineering  degree,  training  in  systems 
and  computer  resource  acquisition,  and  some  program  office  experience, 
will  provide  one  approach  to  the  problem  of  controlling  computer 
resources  in  weapon  systems. 


Appendix  A:  Directives,  Standards,  and  Regulations 


Department  of  Defense  Directives/Instructions 


DODD 

4120.3 

Defense  Standardization  and  Specification  Program 

DODD 

4155. 1 

Quality  Program 

DODD 

4155.19 

NATO  Quality  Assurance 

DODD 

5000.1 

Major  System  Acquisition 

DODD 

5000.2 

Major  System  Acquisition  Process 

DODD 

5000.3 

Test  and  Evaluation 

DODD 

5000.29 

Management  of  Computer  Resources  in  Major  Defense 
Systems 

DODD 

5000. 31 

Interim  List  of  DoD  Approved  High  Order  Programming 
Languages  (H0L) 

DODD 

5000. 35 

Defense  Acquisition  Regulatory  System 

DODL 

5010.12 

Acquisition  Management  Systems  and  Data  Requirements 
Control  List  (AMSDL) 

DODD 

5010.19 

Configuration  Management 

DODD 

5200.28 

Security  Requirement  for  Automatic  Data  Processing 
(ADP)  Systems 

DODI 

7041. 3 

Economic  Analysis  and  Program  Evaluation  for  Resource 
Management 

Standards  and  Specifications 

MIL-STD-109B  Quality  Assurance  Terms  and  Definitions 

D0D-STD-480A  Configuration  Control-Engineering  Changes,  Deviations 

and  Waivers 

MIL-STD-481A  Configuration  Control-Engineering  Changes  (Short 

Form) 


MIL-STD-482A 


Configuration  Status  Accounting  Data  Elements  and 
Related  Features 


MIL-STD-483A 


Configuration  Management  Practices  for  Systems, 
Equipment,  Munitions,  and  Computer  Programs 


MIL-STD-490A 
MIL- STD- 49 9 A 
MIL-STD- 721C 

MIL-STD-881A 
MIL- STD- 152 IB 


MIL-STD-1535A 
MI L- STD- 15 53 B 

MIL-STD-1589C 

MIL- STD- 17 50 A 

MIL-STD-1760A 

MIL-STD-1815A 

MIL-STD-1862B 

D0D-STD-2167 

MIL-Q-9858A 

MIL-S- 52779A 

MIL-S-83490 

MIL-HDBK-334 


Specification  Practices 
Engineering  Management 

Definitions  of  Effectiveness  Terms  for  Reliability, 
Maintainability,  Human  Factors  and  Safety 

Work  Breakdown  Structures  for  Defense  Material  Items 

Technical  Reviews  and  Audits  for  Systems,  Equipment 
and  Computer  Programs 

Supplier  Quality  Assurance  Program  Requirements 

Aircraft  Internal  Time  Division  Command/Response 
Multiplex  Data  Bus 

JOVIAL  (  J73) 

Airborne  Computer  Instruction  Set  Architecture 

Aircraft/Store  Electrical  Interconnection  System 

ADA  Programming  Language 

NEBULA  Instruction  Set  Architecture 

Defense  System  Software  Development 

Quality  Program  Requirements 

Quality  Assurance  Program  Requirements 

Specifications,  Types  and  Forms 

Evaluation  of  a  Contractor's  Software  Quality 
Assurance  Program 


Regulations , 
AFR  70-1 

AFR  70-18 
AFR  80-14 
AFR  700-1 


Manuals,  and  Pamphlets 

Procurement  of  AF  Assigned  Items  Under  the  DoD 
Coordinated  Procurement  Program 

Local  Purchase  Program  (AFSC  Supplement) 

Test  and  Evaluation 

Managing  Air  Force  Communications  Computer  Systems 


AFR  700-2 


Information  Systems  Planning 


AFR  700- J 

AFR  700-4,  Vol  I 
AFR  700-4,  Vol  II 

AFR  800-2 
AFR  800-3 
AFR  800-12 
AFR  800-14 
AFR  800-17 

AFR  800-28 
AFLCR  66-27 
AFLCR  66-37 
AFCMDR  70-16 
AFCMDR  70-24 
AFCMDR  178-1 
AFCMDR  800-1 

AFCMDR  800-3 
AFSCR/AFLCR  80-17 

AFSCM/AFLCM  375-7 

AFSCP  800-14 
AFSCR  800-14 


Information  Systems  Program  Management  and 
Acquisition 

Information  Systems  Program  Management 

Information  Systems  Acquisition  and  Major  Automated 
Information  Systems  Review  Requirements 

Acquisition  Program  Management 

Engineering  for  Defense  Systems 

Acquisition  of  Support  Equipment 

Lifecycle  Management  of  Computer  Resources  in  Systems 

Work  Breakdown  Structure  (WBS)  for  Defense  Materiel 
Items 

Air  Force  Policy  on  Avionics  Acquisition  and  Support 
Automated  Support  of  ATE  Software 
Management  of  Automated  Test  Systems 
Supporting  Contract  Administration 
Subcontract  Management  Program 

Contractor  Management  System  Evaluation  Program 

Acquisition  Management-Contract  Management 
Engineering 

Embedded  Computer  Resources 

AF  Engineering  Responsibility  for  Systems  and 
Equipment 

Configuration  Management  for  Systems,  Equipment, 
Munitions,  and  Computer  Programs 

Air  Force  Systems  Command  Software  Quality  Indicators 
Management  of  Computer  Resources  in  Systems 


AFSCP  800-43 


Air  Force  Systems  Command  Software  Management 
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Appendix  B:  Sample  Cover  Letter  and  Interview  Guides 


DEPARTMENT  OF  THE  AIR  FORCE 

AlA  UNIVERSITY 

AIR  FORCE  INSTITUTE  OF  TECHNOLOGY 
WRIGHT-PATTERSON  AIR  FORCE  3ASc  OH  45433-6S33 


L3Y  (Capt  Roger  Davis,  255-4845)  I  1  MAY  1937 

Software  Personnel  Requirements  Research  Interview 


ASD/ 


1.  Lieutenant  Cynthia  L.A.  Norman,  a  master's  degree  candidate  in  the 
Systems  Management  Program  at  the  Air  Force  Institute  of  Technology 
(AFIT),  is  conducting  research  on  the  education,  skills,  and  experience 
requirements  for  software  management  personnel  assigned  to  program 
offices.  Upon  completion,  she  will  be  able  to  provide  recommendations 
for  software  personnel  support  in  the  program  offices. 

2.  To  achieve  her  objectives,  Lt  Norman  needs  to  interview  someone  from 
program  management,  as  well  as  individuals  in  your  organization  who 
provide  software  management  support  on  your  program.  Attachments  1  and  2 
list  the  information  she  would  like  from  you  and  your  software  people, 
respectively.  Could  Lt  Norman  interview  you  or  your  deputy  and  two  other 
qualified  software  people  in  your  organization?  The  interviews  should 
take  about  30  minutes  for  each  parson.  If  you  are  willing  to  help,  would 
you  please  provide  the  names  and  phone  numbers  back  to  me  by  1  June  1987. 
Lt  Norman  will  then  contact  these  individuals  directly  to  set  up  an 
appointment.  In  addition  to  the  interview,  if  you  could  provide  a  copy 
of  your  organizational  chart,  it  would  help  her  research  efforts. 


3.  Thank  you  for  your  help  in  this  effort.  If  you  have  any  questions, 
feel  free  to  contact  Lt  Norman  at  56569  or  her  thesis  advisor,  Capt  Roger 
Davis,  AFIT/LSY. 


DUMOND,  Lt  Col,  USAF 
.Head,  Department  of  System 
Acquisition  Management 
School  of  Systems  and  Logistics 


2  Atchs 

1.  Program  Manager 
Interview  Guide 

2.  Software  Personnel 
Interview  Guide 


STOCMOTM  THROUGH  KMOWUDOI 


Interview  Guide 
Program  Managers 


Background 

1 .  Name . 

2.  Position. 

3.  Program. 

4.  How  many  years  have  you  been  assigned  to  this  program? 


Software  Costs 


5. 

What 

do 

you 

estimate  are 

your 

software 

costs 

for 

your 

program? 

6. 

What 

is 

the 

total  cost  of 

your 

program? 

7. 

What 

do 

you 

do  to  ensure 

that 

software 

costs 

are 

kept 

to  a  minimum? 

Staffing  and  Organization 

8.  Do  you  have  software  personnel  in  your  organization?  If  not,  do  you 
need  software  personnel?  Who  do  you  have  doing  the  job  of  the 
software  personnel  in  the  mean  time? 

9.  Where  do  you  use  your  software  personnel  in  your  organization? 

10.  How  many  people  are  permanently  assigned  as  software  personnel? 

11.  How  many  people,  in  total,  are  assigned  to  the  organization? 


Duties  and  Responsibilities 

12.  What  are  the  duties  of  your  software  personnel? 

13.  Who  reviews  your  software  technical  documents? 

Documents  may  include: 

a.  Technical  Proposal 

b.  System  specifications  and/or  requirements 

c.  Computer  resource  utilization  reports 

d.  Software  development  manning  profiles 

e.  Engineering  change  proposals 

f.  Development,  test,  and  integration  schedules 

g.  Cost  reports  (CPR  or  C/SSR) 
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14.  What  type  of  procedures  do  you  have  to  ensure  adequate  review? 

15.  Who  conducts  the  reviews  and  audits  of  the  software? 


Education  and  Training 

16.  What  type  of  educational  background  do  you  require  of  your  software 
personnel? 

17.  What  type  of  training  program  do  you  have  for  your  software 
personnel? 

Conclusion 

18.  What  else  would  you  like  to  add? 

19.  What  questions  may  I  answer  for  you? 

20.  Do  you  have  any  additional  materials  that  might  help  in  my  research 


Interview  Guide 
Software  Personnel 


Background 

1 .  Name . 

2.  Position. 

3.  Program. 

4.  How  many  years  have  you  been  assigned  to  this  program? 

Work  Experience 

5.  What  are  your  duties  in  this  program? 

6.  How  much  knowledge  in  software  engineering  is  needed  to 
accomplish  your  responsibilities? 

7.  What  other  programs  have  you  worked  on? 

8.  How  much  do  you  rely  on  your  previous  work  experience  in  your 
current  job? 

Education  and  Training 

9.  What  type  of  educational  background  do  you  have? 

10.  How  has  this  education  been  beneficial  to  your  job? 

11.  What  type  of  formal  training  have  you  had  in  connection  with 
your  job? 

12.  How  has  this  training  been  beneficial  to  your  job? 

Personal  Views 

13.  What  type  of  training  do  you  feel  you  need  but  have  not  been 
able  to  get? 

14.  What  type  of  education  do  you  recommend  for  your  job? 

15.  What  type  of  previous  work  experience  do  you  recommend? 

16.  What  type  of  training  programs  would  you  recommend? 


Conclusion 


"V 

> 


.vV 


17. 

18. 
19. 


What  else  would  you  like  to  add? 

What  questions  may  I  answer  for  you? 

Do  you  have  any  additional  materials  that  might  help  in  my 
research? 


Appendix  C:  The  Acquisition  Life  Cycle 

The  acquisition  life  cycle  consist  of  four  phases:  concept  explo¬ 
ration,  demonstration  and  validation,  full-scale  development,  and 
production  and  deployment.  Certain  prescribed  activities  take  place 
during  each  phase.  Proper  management  of  hardware  and  software  activities 
during  the  phases  is  important.  This  appendix  addresses  the  software 
portion  of  the  acquisition  life  cycle  and  how  it  relates  to  the  hardware 
acquisition.  (See  Figure  2.) 

Concept  Exploration 

The  first  phase,  concept  exploration,  is  spent  refining  solutions, 
performing  studies,  and  developing  alternatives  for  a  required  opera¬ 
tional  capability  (13:61;  18:12).  Some  of  the  types  of  studies  performed 
during  concept  exploration  include  requirements  refinement,  operational 
concept  analysis,  tradeoff  and  optimization  studies,  feasibility  studies, 
and  risk  analysis. 

During  this  phase,  a  Computer  Resources  Working  Group  (CRW'G)  is 
formed.  The  CRW'G  is  responsible  for  developing  an  initial  Computer 
Resources  Life  Cycle  Management  Plan  (CRLCMP),  evaluating  alternatives 
for  computer  resources  support,  identifying  unique  software  quality 
requirements,  and  evaluating  hardware,  programming  languages,  software 
tools,  and  software  development  approaches  (18:12).  Preliminary  system 
specifications  and  an  initial  Test  and  Evaluation  Master  Plan  (TEMP) 
should  be  available  at  the  end  of  concept  exploration,  and  a  system 
requirements  review  (SRR)  should  be  accomplished  (13:64;  18:12). 
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Figure  2.  Software  Development  Life  Cycle  (9:32) 


Demonstration  and  Validation 


The  objective  of  the  demonstration  and  validation  phase  is  to 
validate  a  choice  of  alternatives  so  that  a  decision  can  be  made  whether 
or  not  to  continue  the  project  (13:64;  18:13).  During  this  phase 
requirements  are  further  broken  down  until  the  entire  system  is  described 
and  engineering  studies  are  done  on  the  requirements  (18:13).  The 
engineering  studies  include  requirements  definition,  interface  defini¬ 
tion,  tradeoff  and  optimization  studies,  feasibility  studies,  risk 
analysis,  and  software  support  studies  (18:13). 

During  this  phase  the  TEMP  and  the  CRLCMP  should  be  updated  to 
reflect  the  refined  requirements  (18:13,14).  The  CRLCMP  should  address 
system  concepts,  system  description,  computer  resource  design, 
organizational  responsibilities,  resources  (to  include  personnel, 
facilities,  training,  hardware,  software,  and  integrated  logistics 
support),  management  practices,  and  schedules  (18:42-45). 

Software  prototypes  may  be  developed  and  demonstrated  during  this 
phase  (13:64;  18:13).  Additionally,  plans  for  software  quality 
evaluation  and  configuration  management  should  be  developed  at  this  time 
(18:14).  A  system  design  review  is  held  and  an  authenticated  system/seg¬ 
ment  specification  should  result  from  this  phase  (13:64,65;  18:14). 


Full-Scale  Development 

The  full-scale  development  phase  is  when  the  system,  subsystems, 
support  systems,  software,  training,  and  facilities  are  designed,  built, 


tested  and  evaluated  (13:65;  18:15).  This  is  the  first  time  during  the 


acquisition  life  cycle  when  all  the  subsystems  are  put  together  to  make  a 
prototype  of  the  end  product.  With  this  prototype,  tests  are  done  and 


preliminary  data  concerning  operations  and  maintenance  can  be  collected. 
The  tests  also  show  whether  or  not  the  subsystems  can  work  together  as 
designed.  Additional  software  requirements  may  also  be  identified  during 
this  phase  (13:65). 

Additional  software  requirements  analysis  is  done,  culminating  in 
the  completion  and  authentication  of  interface  requirments  specifications 
and  software  requirements  specifications  for  each  computer  software 
configuration  item  (13:65).  Harware  configuration  items,  and  their 
associated  software  are  also  authenticated  during  design  reviews  (13:65). 

During  the  full-scale  development  phase,  reviews  are  conducted.  The 
software  development  reviews  include  the  Software  Specification  Review 
(SSR),  the  Preliminary  Design  Review  (PDR),  the  Critical  Design  Review 
( CDR) ,  the  Test  Readiness  Review  (TRR),  the  Functional  and  Physical 
Configuration  Audits  (FCA,  PCA),  and  the  Formal  Qualification  Review 
( FQR )  (18:15).  A  third  party,  not  belonging  to  the  program  office  or  the 
contractor,  should  also  do  an  independent  verification  and  validation  of 
the  software  for  functional  effectiveness  and  technical  sufficiency 
(18:15). 

In  addition  to  reviews  of  the  subsystems,  the  hardware  and  software 
configuration  items  are  integrated  and  tested  to  validate  that  the  system 
meets  requirements  (18:15).  The  tests  are  divided  into  two  parts.  There 
is  the  Development  Test  and  Evaluation  (DT&E)  where  a  technical 
assessment  is  done  to  determine  whether  or  not  the  system  meets  the 
specifications  (18:15).  The  Operational  Test  an'*  Evaluation  (OT&E)  is 
done  to  assess  whether  or  not  the  system  meets  the  operational 
requirements  (18:15). 


•r, 

* 
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The  CRWG  is  responsible  for  updating  the  CRLCMP  to  reflect  changes 
made  in  life  cycle  planning  activities  and  the  program  during  this  phase 
(18:15).  Other  documentation  delivered  in  this  phase  include  a  software 
test  plan,  a  preliminary  computer  system  operator's  manual,  preliminary 
user's  manuals,  a  preliminary  computer  system  design  manual,  a  computer 
resources  integrated  support  document,  software  test  descriptions,  a 
software  programmer's  manual,  and  other  support  manuals  (13:65,66,67). 

Production  and  Deployment 

Sometimes  this  phase  is  broken  into  two  separate  phases:  the 
production  phase  and  the  deployment  phase.  In  this  appendix  it  is 
treated  as  one  phase  with  "two  overlapping  periods"  (13:68).  The 
production  periods  spans  the  time  from  when  production  is  approved  to 
when  the  last  system  is  delivered  by  the  contractor  and  accepted  by  the 
government  (13 : 68 ; 18 : 16) .  The  deployment  period  starts  with  the  delivery 
and  acceptance  of  the  first  system,  and  ends  when  the  last  articles  of 
the  system  are  removed  from  the  operational  inventory  ( 13 : 68 ; 18 : 17 ) . 

Software  items  and  hardware  items  are  reproduced  for  each  separate 
system  during  this  phase  (18:16).  Any  redevelopment  or  changes  in  the 
software  during  this  time  will  follow  the  same  development  life  cycle 
used  during  the  full-scale  development  phase  (13 : 68 ; 18 : 16) .  Software 
systems  operator’s  manuals  and  user's  manuals  should  be  delivered  with 
the  completed  systems.  Finally,  Follow-On  Test  and  Evaluation  (FOT&E)  is 
accomplished  on  the  delivered  systems  to  assess  their  operational 
effectiveness  in  their  deployed  environment  (13:68). 


Conclusion 


Between  each  of  the  four  phases,  the  government  decides  whether  or 
not  to  continue  with  the  program.  These  decisions  are  based  on 
information  gathered  during  the  reviews  and  tests  conducted  during  the 
previous  phase.  Coordination  between  all  government  agencies  and 
contractor  agencies  will  help  to  ensure  that  life  cycle  costs  are  kept  to 
a  minimum  by  keeping  changes  in  the  design  minimal. 


Appendix  D:  Interview  Comments 


This  appendix  contains  some  of  the  comments  considered  interesting 
or  significant.  These  comments  were  given  by  persons  interviewed  and  may 
or  may  not  have  been  direct  answers  to  the  questions  contained  in  the 
interview  guide. 

"One  of  the  biggest  concerns  I  have  ...  is  that  software  has  been 
ignored  for  a  long,  long  time.  We've  got  two  programs  right  now  in 
jeopardy.  One  program  has  been  in  slip  for  two  years  because  of 
software.  Unfortunately,  the  contractor  did  not  put  the  emphasis  on  the 
software  that  should  have  been  placed — the  documentation  became  very, 
very  sloppy — and,  is  now  two  years  late  in  doing  an  FCA/PCA,  and  so  we’re 
in  a  stall  mode.” 

"...  And  I  need  good  training  on  what  software  documentation  is 
really  needed,  .  .  •  lifecycle  management  of  software.  I  really  need 
training  on  that.  .  .  .  Three  week  courses  are  too  long.  You  need  to 
break  it  down  to  around  three  day  increments,  and  it  could  be  in  several 
sessions.  We  need  to  be  able  to  do  it  in  a  way  that  people  could  come 
out  of  their  programs  and  go  to  those  sort  of  things." 

"I  guess  we  gotta  emphasize  and  face  up  to  'how  do  you  test 
software?’  We  don't  know  how  to  do  that — really  don't  know  how  to  do 
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ABSTRACT 


Research  and  development  of  new  weapon  systems  can  be  very 
expensive.  In  addition,  the  maintenance  of  these  new  systems  can  be 
expensive,  adding  to  the  overall  life  cycle  cost  of  the  systems.  Proper 
design  of  the  new  systems  can  lessen  the  maintenance  costs.  However, 
more  and  more,  weapon  systems  are  not  just  a  product  of  hardware 
engineering.  Computer  systems  and  software  help  to  run  the  systems  more 
efficiently  and  make  it  easier  for  humans  to  control  the  sophisticated 
hardware. 

The  Air  Force  Systems  Command  is  currently  the  main  procurer  of  new 
weapons  systems  for  the  Air  Force.  In  particular,  the  Aeronautical 
Systems  Division  (ASD)  buys  aircraft  and  aircraft  systems  for  the  Air 
Force.  Currently,  ASD  has  a  "pool*'  of  engineers  for  program  offices  to 
draw  upon  whenever  hardware  technical  expertise  is  needed,  but  the  pool 
of  software  experts  is  very  small.  Do  the  majority  of  the  program 
offices  need  such  a  pool  of  software  experts,  and  if  they  do,  what  type 
of  educational  and  work  experience  should  these  people  have? 

To  answer  this  question,  the  views  of  people  who  are  in  the  program 
offices  were  collected.  Program  managers  and  software  personnel  were 
asked  to  give  their  views  on  education,  training,  and  work  experience. 

The  overwhelming  response  is  that  the  program  offices  do  need 
software  experts,  and  that  even  though  they  have  someone  in  the  office 
who  is  the  designated  software  person,  the  program  offices  still  do  not 
have  enough  support  in  the  management  of  software.  Additionally,  the 
recommended  educational  background  is  in  a  computer  engineering 
discipline.  This  education  should  be  enhanced  with  training  in  systems 
acquisition  and  computer  resources  acquisition  management.  If  possible, 
many  of  the  program  offices  would  like  their  personnel  to  come  to  the 
program  with  prior  program  office  experience. 
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